fix(screenshot): add the capture origin to the image-to-screen conversion - #613

Problem

Screenshot lists no UI elements, so an agent that works from it takes its click targets from the image, and the response tells it how to turn an image pixel into a screen coordinate. When the image is downscaled, the response says (c46911b, for #82 and #99):

Screenshot Coordinate Scale: 2.0 — image pixels are downscaled; multiply every image pixel coordinate by 2.0 before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (400, 300))

That is right only when image pixel (0, 0) is screen coordinate (0, 0). The image starts at the top-left corner of what was captured:

• region=[left, top, right, bottom]: at (left, top).
• display=[...]: at the corner of the selected displays, (1920, 0) for a monitor right of a 1920x1080 primary.
• A full capture: at the corner of the virtual screen, which is left of or above (0, 0) when a monitor sits left of or above the primary one. ImageGrab.grab(all_screens=True) returns the whole virtual screen, and get_annotated_screenshot() already subtracts that corner (uia.GetVirtualScreenRect()) when it draws the element boxes.

For these captures, a point worked out from the image the way the note says misses by the corner's position: with a 1920-pixel-wide monitor left of the primary, a click aimed at that monitor lands 1920 px to the right, on the primary. Snapshot with use_vision=True prints the same note. A capture that is not downscaled, such as a small region or display=[1] on a 1920x1080 monitor, gets no conversion at all: the response shows Screenshot Region and Coordinate Space: Virtual desktop coordinates, but an image pixel is not a virtual desktop coordinate there.

This is the second item of #453 ("Screenshot coordinate guidance omits the crop origin"). #453 was closed after #473 fixed its Scrape item, and its first item was fixed later by #559. #471 covered this item along with others; its author closed it without it being merged, and the note is unchanged on main.

Reproduction

At 3416c76 on Windows 11 (build 22631) with Python 3.14.9, with mocks only; nothing was captured from the real screen. The Desktop came from Desktop.__new__. Its displays, cursor and get_screenshot() were stand-ins, and uia.GetVirtualScreenRect, get_current_desktop and get_all_desktops were patched. The stand-in capture cut its image out of a synthetic virtual desktop with a 9x9 red "button" at a known screen point, the way the Pillow backend cuts a region out of the virtual screen. The script ran capture_desktop_state() and build_snapshot_response() with the Screenshot tool's arguments, found the button in the returned PNG, and mapped the button's pixel to the screen the way the response's worked example maps (200, 150). Output on main:

== full capture, second monitor left of the primary
Screenshot()
  | Screenshot Original Size: (3840,1080)
  | Screenshot Coordinate Scale: 2.0 — image pixels are downscaled; multiply every image pixel coordinate by 2.0 before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (400, 300))
returned image: 1920x540; button at image pixel (210, 150); button on screen at (-1500, 300)
the worked example's mapping sends image pixel (210, 150) to (420, 300): misses by (1920, 0)
== region capture on a 4K monitor with the documented scale setting
WINDOWS_MCP_SCREENSHOT_SCALE=0.5
Screenshot(region=[2000, 1000, 3000, 1600])
  | Screenshot Original Size: (1000,600)
  | Screenshot Coordinate Scale: 2.0 — image pixels are downscaled; multiply every image pixel coordinate by 2.0 before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (400, 300))
  | Screenshot Region: (2000,1000,3000,1600)
  | Coordinate Space: Virtual desktop coordinates
returned image: 500x300; button at image pixel (200, 150); button on screen at (2400, 1300)
the worked example's mapping sends image pixel (200, 150) to (400, 300): misses by (-2000, -1000)
== display capture of a 2560x1440 monitor right of the primary
Screenshot(display=[1])
  | Screenshot Original Size: (2560,1440)
  | Screenshot Coordinate Scale: 1.333333 — image pixels are downscaled; multiply every image pixel coordinate by 1.333333 before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (267, 200))
  | Screenshot Region: (1920,0,4480,1440)
  | Coordinate Space: Virtual desktop coordinates
returned image: 1920x1080; button at image pixel (435, 450); button on screen at (2500, 600)
the worked example's mapping sends image pixel (435, 450) to (580, 600): misses by (-1920, 0)
== region capture at full size (the example in #453)
Screenshot(region=[100, 100, 900, 750])
  | Screenshot Size: (800,650)
  | Screenshot Region: (100,100,900,750)
  | Coordinate Space: Virtual desktop coordinates
returned image: 800x650; button at image pixel (200, 150); button on screen at (300, 250)
the response gives no image-to-screen conversion
== full capture of a single 4K monitor (image starts at (0,0))
Screenshot()
  | Screenshot Original Size: (3840,2160)
  | Screenshot Coordinate Scale: 2.0 — image pixels are downscaled; multiply every image pixel coordinate by 2.0 before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (400, 300))
returned image: 1920x1080; button at image pixel (1200, 650); button on screen at (2400, 1300)
the worked example's mapping sends image pixel (1200, 650) to (2400, 1300): on the button

The README suggests WINDOWS_MCP_SCREENSHOT_SCALE ("0.5" in its example) on 1440p and 4K displays; with it, every region and display capture is downscaled and gets the note.

Fix

• tools/_snapshot_helpers.py: a new _image_origin() returns the screen coordinate of the image's top-left pixel: the corner of screenshot_region for a region or display capture, else the top-left corner of the displays the response lists under Visible Displays, which is the corner of the virtual screen.
• When that corner is not (0, 0), the Coordinate Scale note adds it, in the instruction and in the worked example: multiply every image pixel coordinate by 2.0, then add (-1920,0), the screen coordinate of image pixel (0, 0), before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (-1520, 300)).
• A capture that is not downscaled and does not start at (0, 0) gets a line of its own: Screenshot Origin: (100,100) — image pixel (0, 0) is screen coordinate (100,100); add (100,100) to every image pixel coordinate before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (300, 250)).
• The Screenshot description (tools/snapshot.py, and the same sentence in manifest.json) said to multiply by the ratio of original size to displayed size "to get the actual screen coordinates". It now says that the image starts at the top-left corner of the captured area, not necessarily at screen (0, 0), and to add the screen coordinate of image pixel (0, 0) when the response gives one.
• Unchanged: every capture whose image starts at (0, 0), which includes every full capture on a single monitor. Its note reads word for word as before, and a full-size one still gets no conversion line. The other metadata lines are unchanged too.

With the change, the reproduction prints the same output except for these lines, which change or are added:

== full capture, second monitor left of the primary
  | Screenshot Coordinate Scale: 2.0 — image pixels are downscaled; multiply every image pixel coordinate by 2.0, then add (-1920,0), the screen coordinate of image pixel (0, 0), before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (-1520, 300))
the worked example's mapping sends image pixel (210, 150) to (-1500, 300): on the button
== region capture on a 4K monitor with the documented scale setting
  | Screenshot Coordinate Scale: 2.0 — image pixels are downscaled; multiply every image pixel coordinate by 2.0, then add (2000,1000), the screen coordinate of image pixel (0, 0), before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (2400, 1300))
the worked example's mapping sends image pixel (200, 150) to (2400, 1300): on the button
== display capture of a 2560x1440 monitor right of the primary
  | Screenshot Coordinate Scale: 1.333333 — image pixels are downscaled; multiply every image pixel coordinate by 1.333333, then add (1920,0), the screen coordinate of image pixel (0, 0), before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (2187, 200))
the worked example's mapping sends image pixel (435, 450) to (2500, 600): on the button
== region capture at full size (the example in #453)
  | Screenshot Origin: (100,100) — image pixel (0, 0) is screen coordinate (100,100); add (100,100) to every image pixel coordinate before passing to Click, Move, Scroll, or any loc= argument (e.g. image pixel (200, 150) → screen coordinate (300, 250))
the worked example's mapping sends image pixel (200, 150) to (300, 250): on the button

The single-monitor 4K capture prints exactly what it printed on main.

I took the full capture's corner from the displays already in DesktopState to keep the change in one module. If you would rather have get_state() record the corner it captured (capture_rect, else uia.GetVirtualScreenRect(), as get_annotated_screenshot() does), I can move it there.

Not changed: the note still gives one scale for both axes. #453 also asked for separate X and Y ratios; the truncation in the resize makes them differ from that scale by less than one image pixel across the whole image.

Tests

New tests/test_screenshot_coordinate_origin.py, mocks only. A helper builds a Desktop with __init__ patched out, as in tests/test_snapshot_display_filter.py. Its displays, cursor and get_screenshot() are stand-ins, and uia.GetVirtualScreenRect, get_current_desktop and get_all_desktops are patched. The capture is cut out of a synthetic virtual desktop with a red button at a known screen point. The helper runs capture_desktop_state() and build_snapshot_response() with the Screenshot tool's arguments and decodes the returned PNG. An autouse fixture clears WINDOWS_MCP_SCREENSHOT_SCALE.

• test_image_pixel_maps_to_its_screen_coordinate: finds the button in the returned image, maps its pixel to the screen the way the response's worked example does, and checks that the result is within 2 px of the button. Cases: a full capture with a monitor left of the primary, a full capture with a monitor above it, display=[1] for a 2560x1440 monitor right of the primary, a region with WINDOWS_MCP_SCREENSHOT_SCALE=0.5, and a region at full size. (5 cases)
• test_downscaled_capture_at_screen_origin_keeps_its_note: a full capture of a single 3840x2160 monitor; the note is word for word the one on main.
• test_full_size_capture_at_screen_origin_needs_no_conversion: a full capture of a single 1920x1080 monitor; the response has Screenshot Size: (1920,1080) and no conversion.

I ran each file on its own, in the project's environment:

python -m pytest tests/test_screenshot_coordinate_origin.py -v
python -m pytest tests/test_snapshot_display_filter.py
python -m pytest tests/test_screenshot_grid_lines.py
python -m pytest tests/test_screenshot_window_list.py
python -m pytest tests/test_manifest_tools.py
python -m pytest tests/test_check_versions.py

On main, 5 of the 7 new tests fail. Four fail because the mapped point misses the button: the response maps the button's image pixel to (420, 300), but the button is at (-1500, 300), (700, 680) instead of (700, -400), (580, 599) instead of (2500, 600), and (400, 300) instead of (2400, 1300). The region at full size fails with the response says nothing about turning image pixels into screen coordinates. The two captures that start at (0, 0) pass on both. All 7 pass on this branch.

These pass on main and on this branch with the same counts: tests/test_snapshot_display_filter.py (37 passed), tests/test_screenshot_grid_lines.py (9), tests/test_screenshot_window_list.py (2), tests/test_manifest_tools.py (1) and tests/test_check_versions.py (14). I did not run the whole suite.

ruff check is clean on the three changed Python files. ruff format --check is clean on the new test file. _snapshot_helpers.py and snapshot.py are not formatted on main either (quote style: the dedent() f-strings and name='Snapshot'); this change adds no hunk to their ruff format --diff.

Not tested: a real multi-monitor desktop. Everything ran against stand-ins, and I read the image's starting corner off the capture code (ImageGrab.grab(all_screens=True) and the region/display crops) rather than measuring it on real monitors. To check it on a real machine, place a second monitor left of the main one in Display settings, call Screenshot, pick a point on the left monitor in the image (a window's title text, say), convert it as the response says, and call Move with the result. With this change the pointer should land on that point; on main it lands on the main monitor, one left-monitor width to the right. One case I left alone: if Pillow cannot grab all screens, a full capture falls back to the primary monitor alone (the log says "Failed to capture virtual screen, using primary screen"). That image starts at (0, 0), but get_state() is not told about the fallback, so on a desktop whose virtual screen starts elsewhere the note adds the virtual screen's corner.